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A  neural-network-fiision  architecture  for  automatic  extraction  of  oceanographic  features  from 
satellite  remote  sensing  imagery. 

Farid  Askari  and  Benoit  Zen- 


Executive  Summary:  Satellite  remote  sensing  imagery  provides  the  ASW/MW 
operational  community  with  a  synoptic  view  of  the  prevailing  environmental  conditions 
and  the  capability  to  extract  information  on  oceanic  features/processes  affecting 
acoustic  propagation  conditions.  The  high  data  volumes  and  throughputs  generated  by 
earth  observing  satellites  call  for  the  implementation  of  advanced  information  systems 
including  automatic  pattern  recognition  that  can  aid  the  analyst  in  filtering, 
synthesizing,  monitoring  fast  changing  environments,  and  locating  high-interest  targets 
and  features  more  efficiently.  The  work  reported  here  focuses  on  automatic  detection  of 
oceanographic  features  in  satellite  imagery,  using  artificial  intelligence  techniques.  The 
proposed  architecture  lays  the  foundation  for  multi-sensor  data  fusion  and  extraction  of 
information  for  rapid  environmental  assessment  (REA)  applications.  Future  work 
involves  the  expansion  of  the  number  of  signatures  and  decision  rules  in  the  knowledge 
bank  for  a  more  complete  representation  and  classification  of  tactically  relevant  features 
in  satellite  imagery. 
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A  neural-network-fusion  architecture  for  automatic  extraction  of  oceanographic  features  from 
satellite  remote  sensing  imagery. 

Farid  Askari  and  Benoit  Zerr 


Abstract:  This  report  describes  an  approach  for  automatic  feature  detection  from  fusion 
of  remote  sensing  imagery  using  a  combination  of  neural  network  architecture  and  the 
Dempster-Shafer  (DS)  theory  of  evidence.  Deterministic  or  idealized  shapes  are  used 
to  characterize  surface  signatures  of  oceanic  and  atmospheric  fronts  manifested  in 
satellite  remote  sensing  imagery.  Raw  satellite  images  are  processed  by  a  bank  of  radial 
basis  function  (RBF)  neural  networks  trained  on  idealized  shapes.  The  final 
classification  results  from  the  fusion  of  the  outputs  of  the  separate  RBF.  The  fusion 
mechanism  is  based  on  DS  evidential  reasoning  theory.  The  approach  is  initially  tested 
for  detecting  different  features  on  a  single  sensor  and  extended  to  classifying  features 
observed  by  multiple  sensors. 

Keywords:  automatic  detection,  remote  sensing,  neural  networks,  sensor  fusion. 
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1 

Introduction 


Today,  the  remote  sensing  community  is  faced  with  two  critical  problems.  First 
is  the  problem  of  the  management  of  high  data  volumes  generated  by  the  earth- 
observing  satellites,  and  second  is,  the  integration  of  multiparameter  measurements 
from  different  satellites  and  sensors.  On  both  fronts  the  magnitudes  of  the  problems 
are  expected  to  grow  at  an  accelerated  pace.  By  the  end  of  the  decade  there  will  be  a 
dozen  dedicated  ocean  monitoring  satellites  in  orbit  equipped  with  advanced  sensors 
which  are  capable  of  recording  several  hundred  spectral  and  polarimetric  channels 
simultaneously  over  a  single  resolution  cell  and  with  data  rates  of  over  100  Mbps.  As 
such  a  phenomenon  on  the  ocean  surface  may  produce  numerous  different  sets  of 
responses  from  different  sensors.  There  is  therefore  a  pressing  need  for  more 
advanced  information  systems  which  in  addition  to  high-speed  networking  systems 
and  high  processing  capacity  incorporate  intelligent  and  automatic  pattern 
recognition.  New  quantitative  decision  making  tools  must  be  developed  for  fusing 
and  assimilating  large  volume  multidimensional  data  into  usable  products.  Hence,  the 
objectives  of  the  research  presented  in  this  paper  are  twofold:  1)  Develop  a 
methodology  for  automatic  oceanographic  feature  detection  and  extraction  in  satellite 
remote  sensing  imagery,  2)  Develop  a  system  architecture  for  information  and  sensor 
fusion.  The  role  of  automatic  pattern  recognition  and  feature  extraction  systems  is  to 
assist  the  resource  analyst  in  filtering  and  synthesizing  vast  volumes  of  data  more 
rapidly,  optimizing  decisions,  and  monitoring  fast  changing  environments  more 
quickly. 

Much  published  material  exists  on  the  general  problems  of  automatic  pattern 
recognition,  feature  extraction,  and  combining  multiple  classifiers.  For  automatic 
pattern  recognition  several  classification  algorithms  are  available  including  the 
Baysian  classifier,  k-nearest  neighbor  (k-NN),  distance  classifiers  [1]  and  a  family  of 
neural-network  (NNT)  based  classifiers  [2].  Feature  extraction  schemes  make  use  of 
spatial  information  and  shape  descriptors  for  detection  and  classification.  The 
distinctive  characteristics  of  each  feature  may  be  derived  from  Fourier  descriptors, 
moments,  texture  analysis,  the  Hough  transform  [3]  and  the  two-dimensional  wavelet 
transform  [4].  For  improving  the  overall  classification  accuracy  or  increasing  the 
efficiency  of  the  system,  several  schemes  are  used  for  combining  classifiers  [5,6] 
including  majority  voting,  sum  rule,  product  rule,  fuzzy  integral  [7],  and  the 
Dempster-Shafer  (DS)  formalism  [8,9].  The  primary  focus  of  the  studies  cited  above, 
however,  has  been  on  speech,  handwriting  and  character  recognition  applications. 
With  the  exception  of  the  wavelet  transform  [4],  little  has  been  published  on  the 
applications  of  such  techniques  to  satellite  oceanography. 
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Our  work  introduces  a  new  approach  to  automatic  feature  detection  and  sensor 
fusion  in  remote  sensing  imagery  which  relies  on  a  combination  of  neural  network 
architecture  and  the  DS  theory  of  evidence.  The  use  of  neural  networks  for 
classification  purposes  is  of  course  not  new.  The  approach  reported  here,  however, 
differs  from  previous  investigations.  Most  investigations  using  neural-networks  for 
classification  rely  solely  on  the  outputs  of  the  networks  as  the  discriminator.  That  is, 
after  the  network  is  trained  and  a  generalized  relationship  between  input-output  is 
derived,  new  data  is  classified  as  belonging  to  a  certain  class  by  choosing  the 
maximum  network  output.  Here,  the  network  outputs  are  used  as  inputs  to  a  classifier, 
which  uses  the  information  in  the  framework  of  DS  formalism  to  arrive  at 
classification  results.  In  other  words  the  neural  network  outputs  are  treated  as 
posteriori  probabilities,  with  each  network  supplying  independent  evidence  to  the 
classifier.  Our  approach  also  provides  for  more  flexibility  in  fusing  information  from 
multi-source  images  when  the  sensors  have  different  spatial  resolutions.  Each  image 
is  classified  independently  and  the  final  classification  results  are  transferred  to  a 
geographic  map  in  the  form  of  symbols  or  contours.  Moreover  our  approach  differs 
from  traditional  image  processing  schemes  in  the  manner  in  which  2-D  information  is 
processed.  Here,  we  use  a  template  consisting  of  four  1-D  profiles  taken  at  different 
orientations  over  the  image.  As  such,  the  classification  system  is  capable  of  dealing 
with  shape  information  and  features  containing  2-D  spatial  variability,  without 
complex  and  time  consuming  processing  such  as  2-D  texture  analysis. 

The  article  is  organized  as  follows.  In  Section  2,  we  show  the  various  types  of 
signatures  for  the  oceanographic  features  and  the  rational  for  selecting  the  shape- 
kernels  which  form  the  basis  for  automatic  detection.  We  then  formulate  the  neural 
network  and  fusion  architectures.  The  results  of  applying  the  technique  to  image 
classification  and  feature  detection  are  shown  in  Section  3.  It  is  shown  that  the  system 
is  capable  of  detecting  oceanic  frontal  features  in  satellite  imagery  on  the  basis  of 
training  on  idealized  shapes.  Using  sensor  fusion  and  a  set  of  predefined  rules  we 
develop  a  methodology  for  discriminating  between  wind-induced  versus  sea  surface 
temperatude-induced  roughness  fronts.  Also  a  strategy  for  automatic  detection  of 
salinity  fronts  is  proposed  by  fusing  signatures  from  the  roughness  and  color  fields. 
The  conclusions  are  given  in  Section  4.  The  approach  presented  is  well  suited  to 
rapid,  accurate  and  systematic  search  of  features  in  a  single  image  or  multi-sensor 
data. 
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2 

Materials  and  Methods 


This  section  describes  the  overall  automatic  detection  system.  First,  we  show 
examples  of  oceanographic  features  and  their  corresponding  signatures  that  provide 
the  basis  for  the  automatic  detection  problem.  Descriptions  of  the  neural  network  and 
fusion  architectures  and  a  demonstration  of  the  technique  in  classifying  idealized 
patterns  and  features  follow. 

2.1  Oceanographic  Features  and  Signatures 


When  interpreting  satellite  imagery,  the  human  eye  uses  a  combination  of 
tonal,  spatial  structures  and  textural  features  as  visual  indicators  for  recognizing  or 
distinguishing  features  from  their  backgrounds.  In  order  for  a  machine  to  learn  and 
recognize  features  automatically,  low-level  and  high-level  information  must  be 
furnished  to  the  pattern  recognition  system.  The  low-level  information  consists  of 
feature  attributes  such  as  edges  or  intensity  changes,  lines  or  regions.  The  high-level 
information  consist  of  knowledge  representations  and  inference  mechanisms  which 
describe  the  feature's  physical  attributes.  Tonal  or  intensity  modulations  as  well  as  the 
shapes  are  utilized  as  the  basic  low-level  information  for  feature  extraction. 
Modulation  is  defined  as  the  change  in  local  image  intensity  with  respect  to  the  mean 
background  intensity. 

In  describing  high-level  information,  knowledge  can  be  represented  as 
declarative  or  procedural  [10].  In  declarative  representation,  knowledge  is  assembled 
through  historical  evidence.  In  procedural  representation  a  set  of  predefined  rules 
govern  the  flow  of  information  and  decisions.  The  following  discussion  and  examples 
illustrate  the  manner  in  which  we  utilize  declarative  knowledge  in  formulating  the 
pattern  recognition  system.  Numerous  studies  have  reported  the  relationships  between 
intensity  modulations  in  the  roughness,  color  and  the  temperature  fields  to  physical 
processes  on  the  ocean.  Among  processes  contributing  to  variations  in  the  water 
"color"  in  coastal  regions  are  river  discharge  of  dissolved  organic  matter  and 
suspended  sediment,  re-suspension  of  bottom  material  due  to  tidal  currents,  storms 
and  wave  action,  algal  blooms  and  nutrient  loading  [1 1].  For  small-scale  roughness, 
the  primary  geophysical  processes  that  modulate  the  backscattered  radar  cross-section 
are  temperature  fronts,  discontinuities  in  the  wind  field,  interactions  between  long- 
waves  and  short-waves,  interactions  between  waves  and  currents  and  surfactants  and 
surface  films  [12].  Processes  contributing  to  sea  surface  temperature  anomalies 
include  diurnal  heating  and  cooling,  upwelling,  diffusion  or  advection  by  currents, 
filaments  and  eddies  [13]. 
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Consider  the  variations  in  sea  surface  roughness  as  shown  in  an  ERS-2  SAR 
image  (Fig.  1).  The  1-D  vertical  profile  (Fig.  lb)  across  feature  A  shows  a  distinct 
step-like  (bright-to-dark)  change  in  surface  roughness  with  a  smoother  surface  on  the 
upper  portion  of  the  image.  Historical  evidence  suggests  that  in  SAR  imagery,  two 
primary  physical  mechanisms  contribute  to  step-like  changes  in  sea  surface  roughness 
[14,  15]:  wind  stress  changes  induced  by  the  thermal  stability  variations  near  a  SST 
front  and  wind  stress  changes  induced  by  an  impulsive  atmospheric  wind/gust  front 
[16].  Examples  of  similar  features  and  patterns  appear  in  [16].  A  1-D  horizontal 
profile  across  feature  B  (Fig.  lc)  shows  a  bright  signature  with  enhanced  roughness 
concentrated  over  a  narrow  region.  It  is  widely  accepted  that  narrow-banded 
modulations  are  associated  with  regions  of  strong  wave  current  interactions,  ie, 
convergent  fronts  [16],  when  the  modulation  is  positive  (bright)  and  with  the 
accumulation  of  surfactants,  when  modulations  are  negative  (dark)  [12]. 

In  a  declarative  knowledge  base  containing  step-like  and  Gaussian-shaped 
features  the  modulations  are  as  follows.  For  a  step-like  feature  a  distinct  boundary 
separates  two  regions  with  different  mean  image  intensities,  whereas  for  the 
Gaussian-shaped  feature,  a  pulse  of  finite  width  stands  above  or  below  a  mean 
background.  The  idealized  shapes  provide  an  adequate  description  of  certain  types  of 
oceanic  fronts  in  remote  sensing  imagery.  The  first  type  is  often  associated  with 
boundaries  of  large-scale  currents  and  eddies  observed  in  advanced  very  high 
resolution  (AVHRR)  or  Sea-viewing  Wide  Field-of-view  Sensor  (SeaWIFS)  imagery, 
and  wind  and  sea  surface  temperature  (SST)  fronts  in  SAR  imagery.  And  the  second 
type  is  often  associated  with  river  and  estuarine  plumes,  current  filaments  and  jets  in 
AVHRR  or  SeaWIFS  imagery  and  velocity  fronts  and  slicks  in  SAR  imageiy.  To 
further  generalize  the  problem  and  account  for  the  various  sensor  resolutions,  feature 
length-scales  and  sensor  look-angle  dependencies,  we  construct  pattem/feature 
functions  with  a  broad  range  of  adjustable  parameter.  The  step-like  modulations  are 
further  divided  into  two  subclasses:  steps  with  intensities  increasing  from  left-to- 
right,  and  steps  with  intensities  decreasing  from  right-to-left.  In  addition  to  the 
vertical  steps,  the  steps  are  permitted  to  contain  ramps  with  varying  slopes.  For  the 
Gaussian-shaped  pulses,  which  can  be  either  positive  or  negative,  the  pulse  widths  are 
varied  (Fig.  2). 

Recent  research  has  shown  that  many  ocean  features  produce  anomalous 
signatures  in  more  than  one  field.  Sometimes  a  surface  feature  may  produce 
signature  anomalies  in  all  three  fields,  while  in  others  only  two  of  the  fields  may  be 
expressed.  When  the  fields  do  appear  concurrently  the  question  is  whether  the  fields 
are  collocated  or  is  there  horizontal  dislocation.  There  are  situations  for  example  with 
roughness  and  SST  fronts,  where  one  field  can  lag  or  lead  the  other  depending  on  the 
structure  of  the  atmospheric  boundary  layer  or  the  wind  direction  [14],  It  is  also  likely 
that  when  observing  a  frontal  system,  anomalous  signals  may  emerge  from  different 
positions  within  the  front  because  of  the  differences  in  the  penetration  depths  of  the 
electromagnetic  waves  into  the  surface.  The  IR  and  microwave  energies  are  due  to 
surface  processes,  whereas  the  color  field  results  from  light  upwelled  from  below  the 
surface. 
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The  ultimate  goal  of  fusion  of  the  various  sensors  and  features  is,  however,  to 
arrive  at  a  knowledge  base  that  has  a  higher  information  content  than  the  individual 
parts.  Next,  we  demonstrate  how  the  shapes  can  be  incorporated  in  a  neural  network 
for  automatic  feature  detection. 

2.2  Neural  network  architecture 

Artificial  neural  networks  are  well  suited  to  a  variety  of  problems  in  signal 
and  image  processing  in  which  competing  hypotheses  are  pursued  simultaneously  and 
rapidly  [2],  The  behavior  and  practical  characteristics  in  terms  of  error  rates,  training 
time,  classification  time,  and  memory  requirement  of  several  different  networks  and 
classifiers  are  compared  in  [17].  A  review  of  neural  networks  for  classification  of 
multi-spectral  remotely  sensed  imagery  is  given  in  [18]. 

The  neural  network  architecture  used  here  relies  on  radial  basis  functions 
(RBF)  [2,  19]  for  training  and  pattern  classification.  The  RBF  network  belongs  to  the 
family  of  multilayer  feedforward  neural  networks.  In  network  design,  one  of  the 
important  considerations  in  choosing  the  classifiers  is  how  the  classifier  partitions  the 
feature-space  and  the  shapes  of  its  decision  boundaries.  How  well  a  classifier  can 
generalize  and  discriminate,  new  unseen  data  depends  on  the  structure  of  its  decision 
regions.  The  RBF  classifiers  have  relatively  smooth  decision  boundaries,  and  are  thus 
able  to  generalize  well  to  unseen  data  [17]. 

The  neural  network  (Fig.  3)  used  here  consists  of  three  layers:  input  layer 
containing  the  input  nodes,  hidden  layer  containing  the  basis  function  (BF)  nodes, 
and  the  output  layer  containing  the  output  nodes.  In  general,  a  network  is  trained  or 
tuned  to  recognize  patterns  by  being  shown  a  given  set  of  input-output  pairs.  The 
tunable  parameters  take  the  form  of  weights. 

The  first  step  in  training  involves  the  specification  of  the  number  of  basis 
functions  (BF).  For  our  particular  problem  we  determined  that  five  BF  are  sufficient 
to  achieve  the  desired  classification  performance.  Unit  weights  connect  the  input 
layer  to  the  second  layer. 


The  second  layer  consists  of  a  number  of  multi-dimensional  Gaussian  BF  defined  by: 


y j  =  X  W‘J  exP(  ~cit)  +  bj 


(2.1) 


i=i 


* 


4 
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where  X  =  ( x^,..^)  is  the  input  vector,  and  Y  =  ( y,,  y2,  ...yp)  is  the  output  vector,  C 
is  mean  vector  and  q  is  the  covariance  matrix,  n  is  the  number  of  BF,  and  is  a  bias 
term  which  is  used  to  threshold  the  output  of  the  y'th  neuron  in  the  output  layer.  The 
second  step  in  training  involves  determining  the  BF  centers  and  variances  or  widths 
of  the  BF.  The  RBF  classifier  is  essentially  a  function  mapping  interpolation  method 
that  partitions  the  n-dimensional  measurement  space  into  hyper-volumes  or  regions 
belonging  to  the  separate  classes.  The  RBF  centers  and  widths  define  the  hyper¬ 
volumes.  The  mean  vectors  are  computed  from  the  training  set  by  a  fuzzy  c-means 
clustering  algorithm  [19],  where  the  number  of  clusters  are  equal  to  the  number  of  BF 
needed  by  the  network.  The  variance  is  computed  using  the  nearest  four  neighbors. 

The  output  nodes  ^are  activated  using  a  linear  combination  of  the  BF  nodes: 
zi  =  £  w«yi  +  woj  <2-2) 

i 

where  z-  is  the  output  of  the  y'th  output  node,  yt  is  the  activation  of  the  ith  BF  node  and 
Wy  is  the  weight  connecting  the  ith  BF  node  to  the  y'th  output  node,  and  is  the  bias 
of  the  y'th  output  node.  The  third  step  of  the  training  process  involves  computing  the 
weights,  which  are  determined  using  matrix  pseudo-inverse  approach.  When 
classification  is  exclusively  performed  by  RBF,  an  input  vector  is  classified  as 
belonging  to  the  class  associated  with  the  output  node  with  the  largest  response. 


2.3  Fusion  Mechanism 

In  the  previous  section  we  described  the  architectural  details  of  the  neural 
network.  For  many  applications  it  is  standard  practice  to  use  the  network  as  a  stand¬ 
alone  classifier  and  choose  the  maximum  output  from  a  given  node  for  selecting  a 
particular  class.  Our  goal  here,  however,  is  to  extend  the  technique  into  a 
classification  strategy  for  multiple  information/sensor  fusion. 

We  first  show  the  application  of  the  technique  for  detecting  features  in  a  single 
image/sensor.  A  2-D  image  is  classified  using  four  1-D  profiles.  For  each  1-D  profile, 
the  network  outputs  an  independent  decision  or  belief  that  the  detected  profile 
belongs  to  one  of  the  predefined  classes.  The  individual  opinions  are  then  fused  to 
derive  the  final  classification  results.  The  fusion  mechanism  is  based  on  the  DS 
theory  of  evidential  reasoning  [9, 21].  The  DS  theory  is  a  mathematical  formalism  for 
assigning  beliefs  to  a  set  of  hypothesis,  and  for  combining  belief  in  a  consistent 
manner.  First,  we  illustrate  the  DS  techniques  by  a  simple  numerical  example,  and 
later  show  graphically  the  behavior  the  network/classifier  using  idealized  features. 

The  principal  requirement  for  the  application  of  the  DS  is  the  construction  of 
the  frame  of  discernment  (FOD).  The  elements  of  the  FOD  are  chosen  as  the  possible 
features  that  might  be  encountered  on  a  satellite  image.  We  define  the  FOD  by  the  set 
0  =  {FI,  F2],  where  FI  and  F2  are  two  separate  features.  Contrary  to  a  probability 
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distribution,  which  assigns  belief  to  the  elements  of  FOD,  the  DS  theory  assigns 
belief  to  all  possible  subsets  of  the  FOD,  and  actually  distinguishes  between 
uncertainty  and  ignorance.  The  degree  of  ignorance  is  introduced  by  assigning  belief 
to  the  whole  frame  of  0.  Here,  we  use  the  RBF  outputs  for  assigning  belief. 

For  demonstration  we  extract  a  vertical  and  a  horizontal  profile  over  two 
hypothetical  features.  The  RBF  is  trained  to  distinguish  between  the  two  features 
using  only  two  elementary  profiles,  pi  and  p2.  The  output  of  the  RBFs  for  the  vertical 
and  horizontal  profiles  respectively,  are  {plv,  p2v},  and  {plh,  p2h}.  We  assume  that 
FI  and  F2  are  isotropic  features,  such  that  the  evidence  from  the  horizontal  and 
vertical  profiles  can  be  combined  directly.  The  possible  subsets  of  FOD  are  {FI}, 
{F2},  {FI,  F2},  and  {0}. 

Now  suppose  that  the  RBF  outputs  result  in  the  following  set  of  beliefs:  plv  = 
0.8,  p2v=  0.2,  plh  =  0.55,  p2h  =  0.45.  Here  both  belief  functions,  (plv  >  p2v)  and  (plh 
>  p2h),  support  the  hypothesis  that  feature  FI  is  present,  although  the  support  from 
the  horizontal  profile  is  weaker  than  the  vertical  profile.  Now  if  we  submit  the 
following  set  of  beliefs: 


m,({0})  =  l-plv  =  a  v  =  0.2 

(2.3) 

mi({Fl })  =  plv-p2v  =  b  v  =  0.6 

(2.4) 

m1({Fl,F2})  =  p2v  =  cv  =0.2 

(2.5) 

m2({0  })  =  1  -  plh  =  ^  =  0.45 

(2.6) 

m2({Fl})  =  plh-p2h  =  bh=0.1 

(2.7) 

/n2({Fl,F2})  =  p2h  =  Ch  =  0.45 

(2.8) 

to  the  DS  and  use  the  orthogonal  sum  (m  =  m\  ©  m2)  combination  rule  [21]: 


m{A)  = 


Z»nCAW.<g)  mi<C> 

1-L>n«,m.(g>  “!<C) 


provided  that 


(2.9) 


(2.10) 


The  order  in  which  mass  functions  are  combined  is  irrelevant  because  the  orthogonal 
sum  is  commutative,  mx  ©  m2  =  m2  ©  mx ,  and  associative, 

ml  ©  (m2  ©  m3 )  =  (m,  ©  m2 )  ©  m3 .  Using  2.3-2.8  in  2.9  we  obtain 
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m({Fl})  =  /n1({0}K({Fl})+m2({0})ml({Fl})+m1({Fl})m2({Fl})+.. 

^({FlJKCfFl^JHw^tFl^J^tFl})  (2.11) 

or 

m  ({FI })  =  av  bh  +  at  bv  +  bv  bh  +  bv  Ch  +  Cy  bh  =  0.64  (2.12) 

m({F2})  =  0.0  (2.13) 

We  obtain  a  much  stronger  support  for  feature  FI,  using  the  combined  beliefs 
(m  {Fl}=0.6,  m  {F2}=0.0.  In  this  situation,  DS  concentrates  support  when  there  is 
consent  between  beliefs  and  reduces  it  when  there  is  not.  Next  we  consider  a  situation 
of  conflict  ((ply  >  p2y)  and  (plh  <  p2h),  ie,  when  the  beliefs  don't  agree: 


OTi({0})  =  1-ply  =  a¥  (2.14) 

m1((Fl})  =  plv-p2v  =  bv  (2.15) 

m1({Fl,F2})  =  p2v  =  cv  (2.16) 

and, 

Wl2({0})=  l-p2h  =  ah  (2.17) 

m2({F2})  =  p2h-plh  =  bh  (2.18) 

m2({Fl,F2})  =  plh  =  ch  (2.19) 

Combing  the  above  beliefs  into  DS  and  using  {m-mi  ©  m2)  we  obtain: 

/n({Fl })  =  (by  ah  +  by  ch)/(l-  by  bh)  =  0.574  (2.20) 

m({F2})  =  (av  b„  +  cv  bh)/(l-  bv  bh)  =  0.043  (2.21) 


Here  the  conflict  results  in  the  reduction  of  support  for  both  hypothesis. 


2.4  Pattern  Classification  Using  Idealized  Features 

Now  we  apply  the  technique  to  a  uniform  intensity  image  containing  two 
idealized  features,  a  narrow  pulse  and  a  square  with  intensities  different  from  the 
background  (Fig.  4).  The  2-D  image  is  processed  by  sub-dividing  it  into  a  series  of 
square  blocks.  A  template  containing  four  1-D  profiles  is  used  for  extracting  intensity 
values  along  the  0°  to  180°,  90°  to  270°,  45°  to  225°,  and  135°  to  315°  directions.  In 
every  block,  the  intensity  values  for  each  profile  are  processed  through  a  bank  of 
RBFs.  The  network  outputs,  which  are  regarded  as  a  posteriori  probabilities  are 
transformed  into  beliefs  belonging  to  the  element  of  the  FOD: 

{PULSE,  STEPJLHJH,  STEP_LH_V,  STEP_LH_D1 ,  STEP_LH_D2, 
STEP_HL_H,  STEP_HL_V,  STEP_HL_D1,  STEP_HL_D2} 
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Where  the  following  notation  applies: 

LH:  low-high  transition, 

HL:  high-low  transition, 

H:  horizontal  profile  (0- 1 80), 

V:  vertical  profile  (90-270), 

Dl:  main  diagonal  (45-225), 

D2:  second  diagonal  (135-315), 

PULSE:  used  the  Gaussion  profiles  for  the  4  profiles  equivalently. 

STEP_LH_H:  uses  mainly  the  low-high  step  from  the  horizontal  profile  but 
also  the  low-high  step  from  main  and  second  diagonals. 

STEP_XX_X  are  defined  in  the  same  way. 

Figures  5-8  show  the  system  response  when  the  template  is  placed  in  different 
positions  within  the  2-D  image.  The  network  output  consists  of  a  3x4  matrix,  where 
the  first  dimension  corresponds  to  feature  types  (STEP_LH,  STEP_HL,  Pulse),  and 
the  second  dimension  is  the  profile  orientation  (H,  V,  Dl,  D2).  Figures  5  and  6 
illustrate  the  network  response  when  the  template  is  aligned  respectively,  with  the 
square's  horizontal  and  vertical  edges.  When  positioned  along  the  vertical  edge,  the 
network  shows  consent  between  the  H  and  Dl-profiles,  disagreement  between  the  H 
or  Dl  with  the  D2  profile,  and  no  opinion  is  given  by  the  V-profile.  On  the  other  hand 
when  the  template  is  placed  along  the  horizontal  edge,  there  is  consensus  among  the 
V,  Dl,  and  D2  profiles,  and  no  opinion  is  given  by  the  H-profile.  In  the  final  analysis, 
in  both  cases  the  combined  evidence  supports  the  proposition  that  a  step-like  feature 
resides  at  the  location. 

Next  we  consider  the  case  where  the  template  is  placed  on  the  pulse,  and 
align  Dl  parallel  to  the  pulse's  main  axis.  Here,  the  network  response  (Fig.  7) 
shows  consent  among  the  H,  V,  and  D2  profiles,  with  the  Dl  profile  providing  no 
opinion.  The  combined  vote  is  in  favor  of  the  pulse  (Fig  7c).  Finally,  we  place  the 
template  over  a  region  of  uniform  intensity  (Fig.  8).  Here  all  four  profiles  respond 
unanimously  to  the  background  noise,  and  no  strong  preference  is  given  to  either 
the  pulse  or  to  the  step.  These  examples  clearly  show  that  a  numerical  description 
derived  from  one  or  even  two  profiles  is  not  adequate  for  classification,  but  the 
fusion  of  information  from  four  profiles  do  provide  sufficient  discriminatory 
power  for  classifying  features  with  2-D  variability. 
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Results 


In  this  section  we  first  demonstrate  the  automatic  detection/classification 
capabilities  of  the  system  using  individual  images,  and  then  show  the  multi-sensor 
fusion  results.  For  testing  the  automatic  classification  three  types  of  satellite  imagery 
are  considered:  SAR/ERS2  image,  AVHRR/NOAA-14  image,  and 
SeaWIFS/ORB  VIEW-2. 

In  the  classification  stage,  the  pattern  recognition  system  replaces  the  visual 
interpretation  step  with  quantitative  decision  making.  The  outputs  from  the  classifiers 
are  essentially  thematic  maps,  in  which  the  pixels  in  the  original  imagery  are  classified 
into  one  of  the  several  classes  (or  themes).  The  classes  in  this  case  are  the  predefined 
shapes  (steps,  pulses)  and  the  outputs  correspond  to  the  probability  of  finding  a  class 
at  that  pixel  location.  The  (0,1)  interval  in  each  image  is  mapped  into  to  the  grey  scale. 

We  next  present  the  individual  classification  results.  The  SAR  image  (Fig.  9a) 
shows  two  dominant  plateau  regions  where  mean  intensities  (or  roughness)  change  by 
about  1.6  dB  in  a  step-like  fashion.  Also  visible,  are  two  prominent  narrow-bright- 
curvilinear  structures  oriented  northwest  to  southeast  within  the  low-roughness  region 
in  the  northern  half  of  the  image.  These  pulse-shaped  features  have  modulation 
amplitudes  of  1  and  1.6  dB.  Figures  9b  and  9c  show  respectively,  the  classification 
results  when  the  SAR  image  is  processed  through  the  pattern  recognition  network  for 
detection  of  steps,  and  Gaussian-shaped  pulses.  In  addition  to  detecting  the  primary 
step  that  separated  the  plateau  regions,  the  network  detected  (Fig.  9b)  several 
secondary  steps  in  the  lower  half  of  the  image,  although  the  amplitudes  of  these  steps 
are  much  smaller  than  the  main  step.  Concerning  the  detection  of  pulses,  the  network 
was  able  to  identify  the  curvilinear  structures,  but  in  addition,  it  recognized  numerous 
bright  points  not  easily  visible  in  the  original  image  (Fig.  9c). 


Figures  lOa-c  and  show  respectively  the  SAR,  AVHRR  and  SeaWIFS  imagery. 
The  SST  field  shows  (Fig.  10b)  a  distinct  front  separating  cooler  (by  1  to  1.5  C°) 
coastal  water  from  waters  offshore.  The  SeaWIFS  image  (Fig.  10c)  intensities  shown 
here  correspond  to  variations  in  k532  ‘,  where  k532  is  the  attenuation  coefficient  for  green 
light.  The  important  observation  is  the  horizontal  variability  of  the  k532  ‘  parameter 
which  shows  several  step-like  signatures  associated  with  different  water  masses  in  the 
region. 
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Next  we  classify  the  temperature  and  color  fields  using  the  same  procedure 
applied  to  the  SAR  imagery  and  then  proceed  with  the  fusion  problem.  The  task  of 
fusing  information  over  the  same  region/feature  from  three  separate  sensors  raised  two 
problems.  The  first  problem  resulted  from  the  differences  in  repeat  cycles  of  different 
satellites,  as  well  as  cloud  cover  which  limited  our  data  base  to  only  two  days,  during 
which  all  three  sensors  imaged  the  same  region  near-simultaneously.  The  AVHRR 
image  was  taken  almost  seven  hours  prior  to  the  ERS-2/SAR  pass,  while  the  SeaWlFS 
pass  occurred  approximately  two  hours  after  the  ERS-2/SAR  pass.  The  second 
problem  was  the  registration  of  the  various  images  having  different  spatial  resolutions. 
Instead  of  registering  the  various  images  on  a  pixel-by-pixel  basis,  which  would  have 
resulted  in  loss  of  information  for  the  high-resolution  SAR  image;  we  projected  the 
coordinates  of  the  classification  labels  from  the  low-resolution  imagery  onto  the  geo- 
referenced  SAR  image.  In  Fig.  1 1  yellow  circles  represent  the  positions  of  step-like 
features  in  the  ocean-color  field,  and  red  circles  representing  the  SST  field. 


We  now  turn  to  the  sensor-fusion  problem  and  pose  certain  hypotheses  using 
declarative  and  procedural  knowledge.  For  example  using  a  single  sensor,  ie,  a  SAR 
we  are  able  to  discriminate  between  roughness  fronts  the  signatures  of  which  are 
linked  to  hydrodynamic  processes  versus  wind  stress  variations,  ie,  pulse-shaped  or 
step-like  signatures.  There  are  situations,  however,  in  which  a  single  sensor  will  give 
ambiguous  results  due  to  the  similarities  of  the  feature  signatures.  Such  a  situation 
arises  with  the  step-like  front  observed  in  Fig  10a.  Given  only  the  roughness 
signature,  it  is  unclear  whether  the  signature  is  of  SST  or  wind  origin  and  additional 
physical  evidence  is  required.  If  we  utilize  the  AVHRR  sensor  as  supplementary 
evidence  and  the  following  set  of  elements: 

SST  Front:  SSTF  =  {pls,  p1A}  and  Wind  Front:  WF  =  {p2S,  pM}  (3.1) 

Where: 

pis :  probability  of  having  a  step-type  profile  from  SAR  sensor. 

pjs :  probability  of  having  a  uniform  profile  from  SAR  sensor. 

p1A  :  probability  of  having  a  step-type  profile  from  AVHRR  sensor. 

p2A  :  probability  of  having  a  uniform  profile  from  AVHRR  sensor. 

S,  and  S2  are  the  SAR-roughness  responses,  and  A,  and  A2  are  the  AVHRR-SST 
responses  respectively,  for  the  two  features,  the  pattern  recognition  system  (Fig.  12) 
can  form  a  hypothesis  based  on: 

•  Rulel:  the  feature  is  a  SSTF  if  S,  and  A,  are  both  steps. 

Or, 

•  Rule  2:  the  feature  is  a  WF  if  S2  is  a  step  while  A2  is  a  constant. 
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In  SAR  imagery  a  step-type  profile  response  occurs  for  both  SSTF  and  WF. 
Thus  with  p)S  >  p2S  support  must  be  given  to  both  fronts,  and  one  cannot  discriminate 
between  the  two  fronts.  So  the  FOD  is 


m,({WINF,  SSTF})  =  pis  (3.2) 

«i({0})=l-pIS  (3-3) 


When  p2S  >  pls  support  is  given  to  "other"  non-step  phenomena  (with  uniform 
response)  observed  in  SAR  imagery: 

/Wj({  other})  =  p^  (3.4) 

m1({0})=l-p2S  (3.5) 

With  the  addition  of  the  AVHRR  sensor  if  p1A  >  p2A,  then  support  is  given  to 

SSTF: 


m2({SSTF})=  p1A-p2A  (3.6) 

w2({0})=  l-p1A+p2A  (3.7) 

Whereas  if  p2A  >  p1A  support  is  given  to  WF.  However,  since  it  is  likely  to  have 
other  types  of  phenomena  (non-step)  in  AVHRR,  support  must  also  be  given  to 
"other": 

ot2({WINF,  other})  =  p2A-  p1A  (3.8) 

m2({0})=  l-p2A+p1A  (3.9) 


As  before  ml  and  m2  can  be  combined  using  m  =  m,  ©  m2  to  arrive  at 
individual  mass  probabilities  using  the  fused  results.  The  underlying  assumption  in 
rule  1  is  that  time-scales  involved  in  generating  SST  changes  by  an  atmospheric  wind 
front  which  in  turn  would  induce  roughness  changes  are  generally  much  longer 
(several  hours)  than  generating  purely  wind-driven  roughness  changes  (seconds). 
Given  that  there  is  good  spatial  correlation  between  the  step-like  roughness  front  and 
the  AVHRR-derived  SST  front,  we  can  conclude  that  the  step-like  roughness  front  is 
associated  with  a  SST  front.  We  attribute  some  of  the  dislocation  in  the  northwest 
region  of  the  image  where  the  two  fronts  begin  to  diverge  (Fig.  11)  to  advection 
during  the  seven  hour  period. 

The  other  remarkable  outcome  of  the  fusion  result  is  (Fig.  1 1)  the  near-perfect 
co-registration  of  the  bright-narrow  radar  bands  (Gaussian-shaped  pulse)  and  step-like 
changes  in  the  color  field,  which  we  suspect  to  be  associated  with  a  salinity  front. 
Similar  behaviors  have  been  noted  for  salinity  fronts  observed  on  the  continental  shelf 
of  the  eastern  U.S.  by  [22,  23].  The  bright  radar  signatures  are  attributed  to  strong 
current  convergence  (driven  by  density  anomalies),  and  small-scale  wave  breaking  in 
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the  frontal  zone.  Color  anomalies  are  attributed  to  demarcation  of  water  masses 
having  different  salinities  and  optical  properties.  If  this  behavior  appears  to  be  a 
general  feature  of  salinity  fronts,  the  combination  of  SAR  and  SeaWIFS  would  make 
a  powerful  detection  tool. 
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4 

Conclusions 


In  this  report  we  describe  a  neural-network-based  classification  system  for 
automatic  pattern  recognition.  The  study  has  yielded  the  following  conclusions: 

1.  Based  on  training  on  idealized  shapes,  the  system  is  capable  of 
automatically  detecting  and  locating  frontal  features  in  satellite  remote  sensing 
imagery. 

2.  Utilizing  a  template  consisting  of  a  series  of  1-D  profiles  provides  an 
efficient  alternative  to  2-D  image-feature  detection  and  processing. 

3.  Use  of  declarative  knowledge  in  conjunction  with  a  set  of  predefined  rules 
provides  a  means  for  discriminating  between  physical  processes  and  ocean  features. 

4.  Working  in  the  framework  of  geo-referenced  data  base  with  either 
symbolic  information  or  image-derived  contours  makes  the  sensor  fusion  problem 
more  manageable. 

The  approach  presented  here  is  well  suited  to  rapid,  accurate  and  systematic 
search  of  features  in  multi-sensor  data.  The  credibility  of  the  methods,  however, 
requires  further  testing  with  expanded  data  set  involving  multi-sensor  signatures  as 
well  as  simultaneous  in  situ  observations.  Future  research  should  focus  on  expansion 
of  the  number  of  features/signatures  in  the  knowledge  bank,  the  multi-sensor  fusion 
rules,  and  the  relationships  between  the  rules  and  real  ocean  processes. 
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Figure  la  C-band  (5.4  GHz)  SAR/ERS2  image  with  200x200  pixels  (50  m  pixel 
spacing). 
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Figure  4  Idealized  features  (step  and 
pulse)  and  the  placement  of 
classification  template  at  various 
locations. 
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Figure  9b  Original  SAR  image. 


Figure  9b  Classification  of  step¬ 
like  features. 


Figure  9c  Classification  of  pulse¬ 
like  features. 
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Figure  10a  SAR/ERS2  image  with 
200x200  pixels  (50  m  pixel  spacing) 
taken  over  the  Gulf  of  Cadiz,  Spain  at 
11:08  UT,  14  February  1998. 


Figure  10b  AVHRR/NOAA-14  image 
with  1.1  km  pixel  spacing  taken  over  the 
Gulf  of  Cadiz,  Spain  at  4:01  UT,  14 
February  1998. 


Figure  10c  SeaWIFS/ORBVIEW-2 
image  with  1.0  km  pixel  spacing  taken 
over  the  Gulf  of  Cadiz,  Spain  at  13:02 
UT,  14  February  1998. 
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Figure  11  High  resolution  SAR  image  with  overlay  of  classification 
results  from  AVHRR  and  SEAWIFS  imagery. 
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Figure  12  Sensor  fusion  architecture. 


-26- 


Document  Data  Sheet 


NATO  UNCLASSIFIED 


Security  Classification 

UNCLASSIFIED 

Project  No. 

014-1 

Document  Serial  No. 

Date  of  Issue 

Total  Pages 

SR-306 

June  1999 

32  pp. 

Author(s) 

Askari,  F.,  Zerr,  B. 


Title 

A  neural-network-fusion  architecture  for  automatic  extraction  of  oceanographic  features  from 
satellite  remote  sensing  imagery 


Abstract 


This  report  describes  an  approach  for  automatic  feature  detection  from  fusion  of  remote  sensing  imagery 
using  a  combination  of  neural  network  architecture  and  the  Dempster-Shafer  (DS)  theory  of  evidence. 
Deterministic  or  idealized  shapes  are  used  to  characterize  surface  signatures  of  oceanic  and  atmospheric 
fronts  manifested  in  satellite  remote  sensing  imagery.  Raw  satellite  images  are  processed  by  a  bank  of 
radial  basis  function  (RBF)  neural  networks  trained  on  idealized  shapes.  The  final  classification  results 
from  the  fusion  of  the  outputs  of  the  separate  RBF.  The  fusion  mechanism  is  based  on  DS  evidential 
reasoning  theory.  The  approach  is  initially  tested  for  detecting  different  features  on  a  single  sensor  and 
extended  to  classifying  features  observed  by  multiple  sensors. 


Keywords 

Malta  Plateau  -  sequence  strategy  -  sea  level  changes  -  Quaternary  sediments  -  seismic  reflection 


Issuing  Organization 

North  Atlantic  Treaty  Organization 

Tel:  +39  0187  527  361 

SACLANT  Undersea  Research  Centre 

Fax:+39  0187  524  600 

Viale  San  Bartolomeo  400, 19138  La  Spezia,  Italy 

E-mail:  library@saclantc.nato.int 

[From  N.  America:  SACLANTCEN 
(New  York)  APO  AE  09613 ] 

NATO  UNCLASSIFIED 


Initial  Distribution  for  Unclassified  SR-306 


Ministries  of  Defence 


DND  Canada  1  0 

CHOD  Denmark  8 

DGA  France  8 

MOD  Germany  1  5 

HNDGS  Greece  1  2 

MARISTAT  Italy  9 

MOD  (Navy)  Netherlands  1  2 

NDRE  Norway  1  0 

MOD  Portugal  5 

MDN  Spain  2 

TDKK  and  DNHO  Turkey  5 

MOD  UK  20 

ONRUSA  34 

NATO  Commands  and  Agencies 

NAMILCOM  2 

SACLANT  3 

CINCEASTLANT/ 

COMNAVNORTHWEST  1 

CINCIBERLANT  1 

CINCWESTLANT  1 

COMASWSTRIKFOR  1 

COMSTRIKFLTANT  1 

COMSUBACLANT  1 

SACLANTREPEUR  1 

SACEUR  2 

CINCNORTHWEST  1 

CINCSOUTH  1 

COMEDCENT  1 

COMMARAIRMED  1 

COMNAVSOUTH  1 

COMSTRIKFORSOUTH  1 

COMSUBMED  1 

NC3A  1 

PAT  1 


Scientific  Committee  of  National 
Representatives 


SCNR  Belgium  1 

SCNR  Canada  1 

SCNR  Denmark  1 

SCNR  Germany  1 

SCNR  Greece  1 

SCNR  Italy  1 

SCNR  Netherlands  2 

SCNR  Norway  1 

SCNR  Portugal  1 

SCNR  Spain  1 

SCNR  Turkey  1 

SCNR  UK  1 

SCNR  USA  2 

French  Delegate  1 

SECGEN  Rep.  SCNR  1 

NAMILCOM  Rep.  SCNR  1 


National  Liaison  Officers 


NLO  Canada  1 

NLO  Denmark  1 

NLO  Germany  1 

NLO  Italy  1 

NLO  Netherlands  1 

NLO  Spain  1 

NLO  UK  1 

NLO  USA  1 


Sub-total 

199 

SACLANTCEN 

30 

Total 

229 

I 


! 


